Week 7 Static Analysis vs Large Language Models

Goal: Compare a traditional static analysis tool (Facebook Infer) with ChatGPT for bug detection on a real-world software project.

Learning Outcomes

By the end of this tutorial, you should be able to:

• Understand how static analysis tools detect bugs without executing programs.
• Run Facebook Infer on a real-world C project.
• Use ChatGPT to analyze source code for potential bugs.
• Compare the capabilities of traditional static analysis tools and Large Language Models (LLMs).
• Evaluate the strengths and limitations of both approaches.

Background

There are two common approaches to finding software bugs without executing a program.

Traditional Static Analysis#

Traditional static analysis tools analyze source code using formal program analysis techniques such as:

• Control Flow Analysis
• Data Flow Analysis
• Abstract Interpretation
• Symbolic Execution

These tools reason about all possible execution paths and can detect many well-defined bug patterns. An popular example is Facebook Infer, which focuses on memory errors, leaks, race conditions, and API issues. Infer is open source.

Traditional static analyzers are generally precise for certain classes of bugs but may produce false positives and usually provide limited explanations.

Large Language Models (LLMs)#

Large Language Models (LLMs), such as ChatGPT, approach bug detection differently.

Rather than using formal program semantics, they rely on knowledge learned from a large corpus of source code and documentation.

LLMs can often:

• explain unfamiliar code,
• identify suspicious logic,
• suggest possible bugs,
• recommend fixes,
• answer follow-up questions.

However, LLMs may also:

• hallucinate bugs,
• overlook subtle memory issues,
• make incorrect assumptions about missing context.

Part 1 – Static Analysis using Facebook Infer

The lighttpd webserver#

In this exercise, we will make use of the lighttpd webserver (pronounced “lighty”), version 1.4.17, as the program for this tutorial. You can get it from the original website. It is about 55,000 lines of code in about 90 files. While somewhat small for this class, some analysis tool licenses have LOC limits or scalability issues, so it was chosen as an indicative compromise.

While not as large or popular as apache, at various points lighttpd has been used by YouTube, xkcd and Wikimedia. Much like apache, old verisons of it have a number of known security vulnerabilities.

The Common Vulnerabilities and Exposures system is one approach for tracking security vulnerabilities. A CVE is basically a formal description, prepared by security experts, of a software bug that has security implications.

There are at least ten CVEs associated with lighttpd 1.4.17 tracked in various lists (such as cvedetails or mitre). For example, CVE-2014-2324 has the description “Multiple directory traversal vulnerabilities in (1) mod_evhost and (2) mod_simple_vhost in lighttpd before 1.4.35 allow remote attackers to read arbitrary files via a .. (dot dot) in the host name, related to request_check_hostname.” You can dig into the information listed in, or linked from, a CVE (or just look at subsequent versions of the program where the bug is fixed!) to track down details. Continuing the above example, mod_evhost refers to source file mod_evhost.c, mod_simple_vhost refers to file mod_simple_vhost.c, and request_check_hostname is in file request.c. You will need such information when evaluating the whether or not a tool finds these security bugs.

Facebook’s Infer#

The Infer tool is a static analyzer — it detects bugs in programs without running them. The primary website is fbinfer.com.

Unfortunately, some versions of Infer can be obnoxious to build and install, despite their handy installation guide.

Run Facebook’s Infer on lighttpd#

Once you have Infer built or downloaded, applying it to lighttpd should be as simple as:

Run Facebook’s Infer on lighttpd#

Step 0 - Create a Working Directory#

mkdir ~/COMP2120/
cd ~/COMP2120/

Step 1 - Download lighttpd-1.4.17 Source Code#

cd ~/COMP2120/
curl -LO https://download.lighttpd.net/lighttpd/releases-1.4.x/lighttpd-1.4.17.tar.gz
tar -xzf lighttpd-1.4.17.tar.gz
# This will create the folder ~/COMP2120/lighttpd-1.4.17

Step 2 - Install Infer (Prebuilt Linux Binary)#

cd ~/COMP2120

# Download Infer prebuilt binary
VERSION=1.3.0
curl -fL \
  "https://github.com/facebook/infer/releases/download/v${VERSION}/infer-linux-x86_64-v${VERSION}.tar.xz" \
  -o "infer-linux-x86_64-v${VERSION}.tar.xz"

# Extract Infer
tar -xJf infer-linux-x86_64-v1.3.0.tar.xz

# Add Infer to the current shell's PATH
export INFER_HOME="$HOME/COMP2120/infer-linux-x86_64-v1.3.0"
export PATH="$INFER_HOME/bin:$PATH"

# Verify the installation
which infer
infer --version

Step 3 - Compile lighttpd-1.4.17 Without Infer (Optional)#

cd ~/COMP2120/lighttpd-1.4.17

./configure
make clean
make -j1

Step 4 - Compile lighttpd-1.4.17 With Infer#

cd ~/COMP2120/lighttpd-1.4.17

./configure
make clean

# Remove any previous Infer reports
rm -rf infer-out

# Run Infer while compiling
infer run -- make -j1

You may see an “ASCII Art” progress bar while Infer is running. Infer should ultimately produce output similar to (but everything is fine if you get very different numbers):

After the analysis completes, Infer will generate the analysis results under:

Found 100 issues (console output truncated to 5, see '/home/ubuntu/hw4/lighttpd-1.4.17/infer-out/report.txt' for the full list)
                      Issue Type(ISSUED_TYPE_ID): #
                      Memory Leak(MEMORY_LEAK_C): 47
                          Dead Store(DEAD_STORE): 26
  Uninitialized Value(PULSE_UNINITIALIZED_VALUE): 25
           Null Dereference(NULLPTR_DEREFERENCE): 2

(The exact numbers may differ depending on your Infer version.)

Detailed reports are generated under:

infer-out/report.txt

Exercise 1

Open

infer-out/report.txt

Choose one reported bug that you find interesting.

Record the following information.

Exercise 2

Open the reported source file.

Read approximately 30–50 lines surrounding the reported location.

Answer the following questions before using ChatGPT.

1. What does this function do?
2. Why do you think Infer reported this issue?
3. Do you believe this is a real bug or a false positive?

Part 2 – Bug Analysis using ChatGPT

ChatGPT#

While ChatGPT can be applied to a program’s source code as a static analysis, it does not make use of traditional program semantics information (e.g., abstract syntax trees, dataflow analyses, etc.). Instead, it uses natural language processing (NLP) techniques to analyze code based on learned patterns and examples. Developers might use ChatGPT to identify potential bugs, clarify unfamiliar code, or get suggestions for improvements. You can access ChatGPT at https://chat.openai.com. In this course, you will use the free version.

Note that you can use any other generative AI assistant you prefer, as long as it is free (i.e., costs no money). This is a course requirement associated with equity between students. If you’d like to use another tool, just name it in your report and use it instead wherever this document references ChatGPT.

For this tutorial, you will use ChatGPT to analyze potential issues in one source file from the project: lighttpd. You will then compare and contrast ChatGPT’s output with the results reported by Infer. To ground this comparison, choose a source file that is flagged by Infer and locate a specific bug reported by Infer. Provide that source code (or a relevant excerpt) to ChatGPT and ask whether it detects any bugs. Your goal is to evaluate how ChatGPT’s response compares to the static analysis report.

Now analyze the same piece of code using ChatGPT.

Copy the function (or roughly 50–100 lines surrounding the reported bug) into ChatGPT.

Use the following prompt.

Prompt 1 – Independent Bug Analysis

You are an experienced software engineer specializing in C systems programming and static analysis.

The following code is extracted from the lighttpd web server.

Please perform a detailed bug analysis.

For each potential issue that you identify:

1. Explain what the code is doing.
2. Identify any potential bugs or suspicious behaviour.
3. Explain why the issue could occur.
4. Estimate your confidence (High / Medium / Low).
5. Suggest a possible fix.
6. Classify the issue into one of the following categories:

- Memory Leak
- Null Pointer Dereference
- Use of Uninitialized Memory
- Dead Store
- Resource Leak
- Logic Error
- Integer Overflow
- Buffer Overflow
- Other

If you believe the code is correct, explain why.

Here is the code:

<PASTE SOURCE CODE HERE>

Prompt 2 – Compare with Infer

After ChatGPT finishes, continue with the following prompt.

Facebook Infer reported the following issue:

<PASTE THE INFER REPORT HERE>

Do you agree with Infer?

Please discuss:

1. Is Infer's report correct?
2. Could this be a false positive?
3. Are there additional bugs that Infer did not report?
4. Explain your reasoning using evidence from the source code.
5. Which tool do you think gives a more useful analysis for this example?

(Optional) Prompt 3 – Code Review Style

Please review this function as if you were conducting a professional code review.

Focus on:

- correctness
- memory safety
- resource management
- defensive programming
- readability
- maintainability

Rank every issue by severity:

- Critical
- High
- Medium
- Low

Finally, rewrite the function with your suggested improvements.

Exercise 3

Record ChatGPT’s findings.

Part 3 – Comparing the Two Approaches

Complete the following comparison table.

Discussion Questions

Q1#

Which tool was easier to use?

Why?

Q2#

Which tool provided more useful explanations?

Provide examples.

Q3#

Did ChatGPT discover any bugs that Infer missed?

If so, do you think they are real bugs?

Q4#

Did Infer report any issues that ChatGPT failed to identify?

Why might this happen?

Q5#

Would you trust ChatGPT to review production code?

Why or why not?

Q6#

If you were working in industry, how would you combine static analysis tools and LLMs in your software development workflow?

Take-away

Traditional static analysis tools and Large Language Models have complementary strengths.

Facebook Infer

Advantages

• Based on formal program analysis.
• Excellent at detecting memory-related defects.
• Consistent and reproducible.
• Suitable for large-scale automated analysis.

Limitations

• Difficult to understand reports.
• Limited explanations.
• Cannot answer follow-up questions.
• May generate false positives.

ChatGPT

Advantages

• Easy to use.
• Provides natural-language explanations.
• Can reason about program logic.
• Suggests possible fixes.
• Supports interactive debugging.

Limitations

• May hallucinate bugs.
• Can miss subtle memory or concurrency issues.
• Results depend heavily on the quality of the prompt.
• Does not perform complete program-wide analysis.

Summary#

Rather than replacing static analysis, LLMs complement traditional tools.

A practical workflow is:

1. Use Infer to automatically detect potential defects.
2. Use ChatGPT to interpret reports, explain root causes, suggest fixes, and assist with debugging.

Combining both approaches often provides a more effective software engineering workflow than using either tool alone.