Questions for MSR 2024 Mining Challenge

[markxiaotao]

Feel free to post any questions here!

[markxiaotao]

Q: What Unicode encoding does the JSON data use?

JSON data is encoded using UTF-8 by default. And the encoding for the content of the code file is Base64

[markxiaotao]

The way we calculated the tokens of prompts or answers is the same as https://stackoverflow.com/a/75804651

import tiktoken

def get_num_tokens_from_string(string: str) -> int:
    encoding = tiktoken.get_encoding("cl100k_base")
    num_tokens = len(encoding.encode(string, disallowed_special=()))
    return num_tokens