By 
By Melissa Cogger and Talita Laubscher, Partners, Bowmans
The use of AI in human resources necessarily involves the processing of personal information – from curricula vitae and interview recordings to biometric data, medical certificates in support of sick leave applications and performance metrics.
RELATED: South Africa: ECA Amendment Bill published for public comment
The Protection of Personal Information Act, 2013 (POPIA) imposes strict obligations on the processing of personal information and provides specific protections pertaining to automated decision-making.
Section 71: Automated decision-making and profiling
Section 71 of POPIA is the provision of most direct relevance to AI-driven HR decisions. It provides that data subjects may not be subject to a decision that results in legal consequences for them or that affects them to a substantial degree, and that is based solely on the automated processing of personal information intended to provide a profile of such person. The section expressly identifies the following categories of profiling: performance at work; creditworthiness; reliability; location; health; personal preferences; and conduct. The list of categories is not closed.
The reference to ‘performance at work’ is particularly significant in the HR context, and so is ‘health’ (in the context of the administration of sick leave and incapacity processes), as well as ‘personal preferences’ insofar as these (through, for example dietary requirements) might reveal the employee’s religious or philosophical beliefs.
Where an AI system generates a performance rating, identifies an employee for promotion or demotion, or recommends that a candidate be rejected for a role, and where that decision has legal consequences or substantially affects the individual, and is solely relied upon, section 71 of POPIA applies.
“Section 71 does, however, provide for certain exceptions”
Section 71 does, however, provide for certain exceptions: automated decision-making is permissible where the decision is taken in connection with the conclusion or execution of a contract and appropriate measures have been taken to protect the data subject’s legitimate interests; or the decision is governed by a law or code of conduct in which appropriate measures are specified for protecting the legitimate interests of data subjects.
In relation to the conclusion or execution of a contract, the measures taken must include providing the data subject with an opportunity to make representations and requiring the responsible party to provide the data subject with sufficient information about the underlying logic of the automated processing to enable the data subject to make such representations.
The responsible party must therefore have sufficient understanding of how the automated decision-making process came to its result, which can be difficult in light of the ‘black box’ problem. The ‘black box problem’ in AI refers to the difficulty in understanding how certain AI systems arrive at their outputs or decisions. The term borrows from engineering, where a ‘black box’ is any system whose internal workings are opaque – one can observe inputs and outputs but cannot see the process in between. The ‘black box’ problem undermines accountability.
“Interrogating when HR decision is based solely on AI profiling exercise”
In the employment context, such as a recruitment or promotion process where AI is used to shortlist candidates, this means that the affected employee or applicant must be afforded the opportunity to be heard and to challenge the outcome of the automated decision-making (ie. the shortlisting), and to engage with a human decision-maker, and must be given insight into how the AI system reached its conclusion if:
- AI is used to take a preliminary decision, which in turn informs the ultimate decision which results in legal consequences; or
- affects the individual to a substantial degree; and
- is based solely on the AI profiling exercise.
As such, it would be important for recruitment agencies or HR personnel relying on AI to shortlist candidates to consider whether the persons excluded have been excluded on grounds that are rational, fair and objectively justifiable so that the decision is not solely based on automated processing.
Even if section 71 of POPIA is not triggered (for example the decision does not have a significant effect on, or legal consequences for, a data subject), the principles of lawful processing and other obligations in POPIA will still apply.
Section 18: Notification requirements
Section 18 of POPIA requires that, at the time personal information is collected, the responsible party must take reasonably practicable steps to ensure that the data subject is aware of, among other things:
- the information being collected and the source from which it is collected (if not from the data subject directly);
- the purpose of the processing;
- whether the supply of information is voluntary or mandatory, and the consequences of failure to provide it;
- any planned transborder flows of personal information; and
- the recipients or categories of recipients of the information.
This notification obligation is particularly pertinent where employers deploy AI tools provided by third-party vendors. Applicants and employees must be informed that their personal information will be processed by an AI system and what the purpose of such processing is.
Conditions for lawful processing
Beyond sections 18 and 71, POPIA’s general conditions for lawful processing impose additional obligations on employers using AI in HR when such AI tools process personal information. The following requirements are of particular relevance:
- Minimality(section 10): Personal information must be adequate, relevant and not excessive in relation to the purpose for which it is processed. AI tools that harvest large volumes of data – for example, scraping social media profiles or analysing keystroke patterns – may offend this principle;
- Processing limitation and purpose specification(sections 13 and 15): Personal information must be collected for a specific, explicitly defined and lawful purpose and must not be further processed in a manner incompatible with that purpose. An employer who collects information for recruitment purposes may not subsequently use it for unrelated algorithmic profiling;
- Security safeguards(section 19): The responsible party must secure the integrity and confidentiality of personal information by taking appropriate, reasonable technical and organisational measures. Where AI tools are cloud-based or provided by third parties, employers must satisfy themselves that adequate security measures are in place; and
- Retention(section 14): Personal information must not be retained longer than necessary for the purpose for which it was collected. Employers must ensure that AI systems do not retain candidate or employee data indefinitely.
Special personal information
Chapter 3 of POPIA restricts the processing of ’special personal information’, which includes information concerning a person’s religious or philosophical beliefs, race or ethnic origin, trade union membership, political persuasion, health or sex life, biometric information and criminal behaviour. The processing of such information is prohibited unless a specific exemption applies (such as the data subject’s consent or where processing is necessary for employment law purposes). Further considerations in terms of prohibited grounds of unfair discrimination should also be taken into account.
AI systems that infer or process such information – for instance, facial recognition software that may identify race or ethnicity, or wellness tools that process health data – require particular scrutiny.
Recruitment agencies: operator vs responsible party
Where a recruitment agency deploys AI tools on behalf of an employer, the distinction between ’responsible party’ and ’operator’ under POPIA becomes critical. The responsible party is the entity that determines the purpose of and means for processing personal information (typically the employer). The operator is the entity that processes personal information on behalf of the responsible party pursuant to a contract or mandate (typically the recruitment agency or AI vendor).
This distinction has important consequences. The responsible party retains ultimate accountability for POPIA compliance, even where processing is outsourced to an operator. Employers must accordingly ensure that their contracts with recruitment agencies and AI vendors impose POPIA-compliant obligations, including data security requirements, restrictions on purpose and retention and compliance with section 71 to the extent it applies.
Employee use of AI systems
A distinct but equally significant risk arises where employees use AI systems in the course of their employment and, in doing so, upload personal information to those systems.
Where an employee inputs personal information (whether of colleagues, customers or third parties) into an open, publicly accessible AI tool, the employer, as responsible party, may be held accountable for a breach of POPIA’s conditions for lawful processing. This is so because the employer determines the purpose and means for processing in the employment context and must secure the integrity and confidentiality of personal information under its control.
To mitigate this risk, employers should implement acceptable AI use policies that clearly define permissible and impermissible uses of AI tools; establish guardrails on what categories of personal information may be uploaded to AI systems (if any); and, where possible, prefer closed or enterprise AI systems that do not expose data to public models over publicly accessible AI platforms, or require that personal information be suitably de-identified. Training and awareness programmes should accompany these measures to ensure that employees understand their obligations under POPIA when interacting with AI systems.
Conclusion
POPIA provides a robust, if not always straightforward, framework for regulating the use of AI in HR. Section 71’s restriction on automated decision-making, combined with the notification requirements of section 18 and the general conditions for lawful processing, impose meaningful constraints on the deployment of AI tools.
Employers must ensure that AI systems are not making consequential decisions without human oversight, that affected individuals are notified of the use of AI in the processing of their personal information and to the extent a decision is solely made on the basis of automated decision-making that results in legal consequences or affects them to a substantial degree, be given an opportunity to make representations (with sufficient information about the underlying logic of the AI system), and that the processing of personal information is minimal, purpose-limited, secure and time-bound.
The use of AI should be considered and incorporated into a responsible party’s compliance framework and personal information impact assessment to ensure that adequate measures and standards exist to ensure compliance with the conditions for lawful processing.
