This tutorial points the hf command and the huggingface_hub Python library at a Virtual Registry that caches models and datasets, from huggingface.co directly or through a JFrog Artifactory HuggingFaceML repository.
huggingface_hub is installed, which provides the hf command.virtual_registry:
registries:
- name: huggingface
remotes:
- url: https://huggingface.co
The examples below reach the registry as https://huggingface.example.com, a host that the registry named huggingface answers (see name). Package types describes how each request is cached.
huggingface_hub reads the address of the Hub from HF_ENDPOINT, and so does each library that downloads through it, such as transformers:
export HF_ENDPOINT=https://huggingface.example.com
hf download prajjwal1/bert-tiny pytorch_model.bin
The library reads the variable when it is imported, so set it before the process starts.
pytorch_model.bin is stored with Xet, and hf_xet, which huggingface_hub installs, downloads such a file with a token from the Hub even when the repository is public. Without the sup-auth addon, the registry passes a request that carries a token on to the Hub without caching it or rewriting the answer, so hf_xet fetches the file from the Hub’s CDN directly.
A gated or private repository needs a token in HF_TOKEN, which the registry passes on to the Hub, and its files are cached only with the same addon.
Give the registry the Artifactory host as its remote:
virtual_registry:
registries:
- name: huggingface
remotes:
- url: https://artifactory.example.com
and include the repository’s path in HF_ENDPOINT:
export HF_ENDPOINT=https://huggingface.example.com/artifactory/api/huggingfaceml/<repo>
This works with the repository’s Xet setting on or off.
With deny_unknown_hosts set, the registry reaches only its remotes and the hosts in extra_known_hosts. The Hub serves file contents from other hosts, which have to be listed:
virtual_registry:
redirects:
deny_unknown_hosts: true
extra_known_hosts:
- us.aws.cdn.hf.co
- cas-server.xethub.hf.co
The Hub can hand out CDN hosts other than these. hf reports a 403 Forbidden for a host that is missing without naming it, and varnishlog on the registry’s host prints redirectz host not allowed: <host>:
varnishlog -d -q 'RespStatus == 403' -i RespReason