ChatGPT can write smart contracts, but developers should avoid using it to audit their own code, researchers said.
Researchers from blockchain security company Salus Security evaluated the smart contract auditing capabilities of GPT-4, OpenAI’s most powerful large language model (LLM). They concluded that although the accuracy of vulnerability detection is high, the recovery rate is dangerously low compared to smart contract audit tools.
In a recently published study, two researchers selected eight sets of smart contracts that were injected with 18 types of vulnerabilities, for a total of 60 different types of vulnerabilities. Their goal was to evaluate whether OpenAI’s LLM could mimic a professional auditor, parsing code and finding vulnerabilities.
Researchers found that GPT-4 detected seven types of vulnerabilities with an accuracy rate of more than 80%. However, across the dataset, recall was very low, around 11%. This suggests that “GPT-4 may miss some vulnerabilities during detection.”
Recall is the proportion of data samples that the model can correctly identify as belonging to a class out of all samples, or the true positive rate. Precision refers to the ratio of true positives to false positives, i.e., how many junk positives are included in the data set.
The researchers noted that their results show that GPT-4’s vulnerability detection capabilities are substandard and should only be used as a supplementary tool for smart contract audits.
“In summary, GPT-4 can be a useful tool to support smart contract audits, especially in code parsing and providing vulnerability hints. However, due to limitations in vulnerability detection, it cannot currently fully replace professional audit tools and experienced auditors,” the researchers concluded.
The study supports other research that has dismissed claims that artificial intelligence (AI) will replace developers. As specialized AI-based tools become increasingly advanced,
Although our ability to write code and detect vulnerabilities is getting better, we still can’t rely on ourselves, especially in the world of digital assets where minor vulnerabilities can expose us.
For artificial intelligence (AI) to function properly within the law and succeed in the face of growing challenges, it must integrate enterprise blockchain systems that ensure data input quality and ownership. This helps keep your data safe while ensuring immutability. data. Check out CoinGeek’s coverage of this emerging technology to learn more about why enterprise blockchain will become the backbone of AI.
WATCH: Does AI know what it’s doing?
Are you new to blockchain? To learn more about blockchain technology, check out CoinGeek’s Blockchain for Beginners section, our ultimate resource guide.