Update Mixtral BS

[yeandy]

Description

Update Mixtral batch size since 256 taking very long time.

Tests

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run one-shot tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.

[vipannalla]

Higher batch_size is supposed to be faster given the same number of samples. How much slower is the current run? and how much faster is the run with batch_size = 128?

[yeandy]

It takes about ~12 min for 128 batch on bf16. Haven't tested for int8.

Not sure why it's hitting timeout at 3 hours for int8 version. Let me try running int8 at 128.

[RissyRan]

Wait! we have already quantized for inference on MaxText?

[yeandy]

Yes!

[RissyRan]

Which implementation is this one? I don't think I enabled quantization for both matmul or megablox yet (I mean MoE block, other blocks are enabled). Or other implementation we are talking about here?

[vipannalla]

I think this configuration is using the old for for loop implementation, which Ranran cleaned up recently. I don't think the current code support quantization right now.