Evaluating Dialogue Act Classifiers under Class Imbalance

Background

Dialogue act tagging identifies the communicative function of an utterance, such as a statement, question, acknowledgement, or signal of non-understanding.

Using the Switchboard Dialogue Act Corpus, I developed and compared neural classifiers for predicting one of 43 dialogue-act categories.

A central challenge was the strong class imbalance in the dataset. Common dialogue acts appeared frequently, while some categories contained relatively few examples.

This created an important evaluation question: does a model with strong overall accuracy also perform reliably across less frequent classes?

Research Goals

1. Build an utterance-level dialogue-act classifier using recurrent neural networks.
2. Measure how class imbalance affects predictions for rare dialogue-act categories.
3. Compare a BiLSTM baseline with a more complex CNN-BiLSTM architecture.
4. Evaluate models using both overall accuracy and class-specific performance.
5. Examine whether an improvement in aggregate accuracy necessarily corresponds to more reliable behaviour across classes.

Methodology

BiLSTM Baseline

The baseline classifier used:

• Token embeddings for utterance representation
• Two stacked bidirectional LSTM layers
• A dense classification layer predicting one of 43 dialogue-act labels
• Cross-entropy loss
• Adam optimisation

Each utterance was classified independently.

Minority-Class Analysis

After evaluating overall performance, I used a confusion matrix to examine two comparatively rare dialogue-act categories:

• br — signal non-understanding
• bf — summarize/reformulate

For these classes, I calculated class-specific recall to measure how often examples belonging to each category were correctly identified.

CNN-BiLSTM Variant

I also implemented an alternative neural architecture combining:

• Parallel CNN filters with kernel sizes 3, 4, and 5 for local textual feature extraction
• Batch normalisation
• Max pooling
• Two stacked bidirectional LSTM layers
• Dense and dropout layers for classification

The purpose of this experiment was to test whether a richer neural representation improved classification performance beyond the BiLSTM baseline.

Results

BiLSTM Baseline

The BiLSTM achieved:

• Test accuracy: 69.71%
• br recall: 44.12%
• bf recall: 1.95%

Viewed only through aggregate accuracy, the model appeared to provide reasonably strong classification performance.

However, class-specific analysis revealed a very different picture. The model correctly identified fewer than half of the br examples and almost completely failed to recognise the bf class.

This demonstrated that aggregate accuracy concealed substantial failure on underrepresented categories.

CNN-BiLSTM

The CNN-BiLSTM achieved:

• Test accuracy: 70.21%
• br recall: 38.24%
• bf recall: 0.98%

Compared with the BiLSTM baseline, overall accuracy increased slightly:

69.71% → 70.21%

However, performance on both minority classes deteriorated:

• br recall: 44.12% → 38.24%
• bf recall: 1.95% → 0.98%

The more complex architecture therefore produced a higher aggregate accuracy while becoming less effective at recognising the two rare classes examined.

The central result was therefore not that one architecture was universally better than the other.

Instead, the experiment showed that the definition of improved performance depends on which errors are considered important.

Contributions

• Built and evaluated a 43-class dialogue-act classification system using the Switchboard corpus.
• Implemented and compared BiLSTM and CNN-BiLSTM neural architectures.
• Achieved approximately 70% held-out test accuracy with both models.
• Used confusion-matrix analysis to identify class-specific failure modes hidden by aggregate performance.
• Quantified the gap between overall model performance and recall on two underrepresented dialogue-act categories.
• Demonstrated that a modest improvement in overall accuracy could coincide with worse minority-class recall.
• Evaluated model quality through the distribution of errors, rather than treating a single aggregate metric as sufficient evidence of improvement.

Limitations

The CNN-BiLSTM experiment should be interpreted as an architectural comparison at the utterance level, rather than as an evaluation of dialogue-history or inter-utterance contextual modelling.

The analysis also focused specifically on two rare dialogue-act categories. A more comprehensive evaluation could examine macro-averaged metrics and class-specific behaviour across all 43 labels.

Impact

The most important result of this project was that a model can improve according to an aggregate metric while becoming worse for particular subsets of the data.

The CNN-BiLSTM achieved slightly higher overall accuracy than the BiLSTM, yet its recall decreased for both minority dialogue acts examined.

This highlighted a broader model-evaluation problem: choosing a model requires deciding which errors matter, rather than assuming that the model with the highest average score is necessarily the better system.

That perspective became relevant to my broader interest in reliable AI and decision-making under uncertainty, where model performance must be understood not only through average predictive accuracy, but also through the conditions and cases in which the model fails.