Full Stack Deep Learning

<aside> 💡 80-90% of time debugging and tuning

</aside>

1. Why is DL troubleshooting so hard?

[Poor Model Performance]

• Implementation bugs
• Hyperparameter choices
• Data/model fit
• Dataset construction
  • Not enough data
  • Class imbalances
  • Noisy labels
  • Train / test from different distributions
  • etc

2. Strategy for DL troubleshooting

Start simple → complexity

[Decision Tree]

2.1 Start Simple

[간단한 아키텍처 선택]

• 데이터가 이미지일 경우, LeNet → ResNet 고려
• 데이터가 시퀀스일 경우, LSTM → Attention 기반 모델이나 WaveNet 고려
• 기타 경우, Fully connected neural net with one hidden layer에서 고급 네트워크 고려

[Sensible defaults 사용]

• Optimizer:

  Adam optimizer with learning rate 3e-4

• Activations:

  relu (FC and Conv models), tanh (LSTMs)

• Initialization

  He et al. normal (relu), Glorot normal (tanh)

• Regularization & Data normalization:

  None

[Inputs 정규화]

• Subtract mean and divide by variance

[문제 단순화]

• Start with a small training set (~10,000 examples)
• Use a fixed number of objects, classes, image size, etc.
• Create a simpler synthetic training set

2.2 Implement & debug