Home Description Explore Download People Sign In Upload History Logout

HAKE: Human Activity Knowledge Engine

MVIG - Shanghai Jiao Tong University


Human Activity Knowledge Engine (HAKE) aims at promoting activity understanding with a large-scale knowledge base and visual reasoning. To afford knowledge-based learning/reasoning, HAKE provides not only 26M+ human body part-level atomic action labels (Part States) but also holistic object knowledge annotations. It boosts the performances of several widely-used activity benchmarks (HICO, HICO-DET, V-COCO, AVA, etc), and can convert human boxes into multi-modal representations for diverse downstream tasks, e.g., image/video action recognition/detection, captioning, VQA, visual reasoning, etc. Now, the HAKE project contains the works of 8 papers (CVPR'18/19/20, NeurIPS'20, TPAMI'21).

We will keep enriching HAKE to make it a general research platform of knowledge mining, visual reasoning, and causal inference. Come and join us!


1) HAKE-Image (CVPR'18/20): HAKE-HICO, HAKE-HICO-DET, HAKE-Large, Extra-40-verbs.

2) HAKE-Action-TF, HAKE-Action-Torch (CVPR'18/19/20, NeurIPS'20, TPAMI'21): SOTA action understanding methods and the corresponding HAKE-enhanced versions (TINIDN).

3) HAKE-3D (CVPR'20): 3D human-object representation for action understanding (DJ-RN).

4) HAKE-Object (CVPR'20, TPAMI'21): object knowledge learner to advance action understanding (SymNet).

5) HAKE-A2V (CVPR'20): Activity2Vec, a general activity feature extractor based on HAKE data, converts a human (box) to a fixed-size vector, PaSta and action scores.

6) Halpe: a joint project under Alphapose and HAKE, full-body human keypoints (body, face, hand, 136 points) of 50,000 HOI images.

7) HOI Learning List: a list of recent HOI (Human-Object Interaction) papers, code, datasets and leaderboard on widely-used benchmarks.


HAKE-A2V (Activity2Vec): [Code], [Result Description] [YouTube] [bilibili]
Pipeline: Images/Videos --> human box + ID + skeleton + part states + action + representation
(Place your mouse to view big pictures)
357 K+
Activity Classes
673 K+
Human Instances
220 K+
Object Instances
345 K+
Instance Actions
Part State Classes
26.4 M+
Human Part States
1 M+
Object States
{{ pageviewNumber }}
{{ uploadNumber }}