Bug 915667 – sci-libs/tokenizers can't be built with rust-1.73

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description Lorenzo Bertini 2023-10-13 06:44:59 UTC]

Dear maintainer,

build fails with the following error:

error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
   --> src/models/bpe/trainer.rs:526:47

This has been noted in this: https://github.com/huggingface/tokenizers/issues/1359, and it's probably due to rust-1.73. I don't have a solution in mind.

Best wishes.

[Comment 1 Sam James 2023-10-14 08:05:10 UTC]

Thanks. Please always include the full build.log and emerge --info.

[Comment 2 Tupone Alfredo 2023-11-15 20:10:39 UTC]

updated to 0.14.0.
Does this bug apply to the new version ?

[Comment 3 Lorenzo Bertini 2023-11-16 13:10:07 UTC]

(In reply to Tupone Alfredo from comment #2)
> updated to 0.14.0.
> Does this bug apply to the new version ?

It does unfortunately. Apparently it was fixed by 0.14.1: https://github.com/huggingface/tokenizers/issues/1050.

Build log: https://bpa.st/E37A
Emerge info: https://bpa.st/L2TA

[Comment 4 Sam James 2023-11-16 15:47:00 UTC]

Please always attach logs, no pastebins.

[Comment 5 Tupone Alfredo 2023-11-16 21:39:30 UTC]

updated to 0.14.1.
Close if fixed.

[Comment 6 Lorenzo Bertini 2023-11-17 20:04:25 UTC]

(In reply to Tupone Alfredo from comment #5)
> updated to 0.14.1.
> Close if fixed.

Fixed. Many thanks. I'll close. Any chance we see transformers updated?

[Comment 7 Tupone Alfredo 2023-12-10 21:44:12 UTC]

sci-libs/transformers updated to a version compatible with tokenizer last.

Please test