Gradient Descent and Convex Intuition TODO
Concept
Gradient descent is an optimization method that exploits the fact that the gradient of the objective function points in the direction of steepest ascent, and repeatedly moves a step of size equal to the learning rate in the opposite direction. A function is convex if the line segment joining any two points in its domain always lies on or above the function's graph; this property guarantees that a local minimum is also the global minimum, and that a point where the gradient is zero is optimal. For a smooth convex function whose gradient is L-Lipschitz continuous, choosing a learning rate at or below 1/L guarantees convergence, and adding strong convexity yields a faster, exponentially decaying error rate. A learning rate that's too large causes divergence, one that's too small is slow, and poor conditioning causes zig-zagging in narrow, elongated valleys — which is why techniques like momentum, adaptive learning rates, and preconditioning are used to mitigate it.
In numerical optimization beyond machine learning — market-making parameter tuning, calibration — you need to tell whether a failure to converge comes from the problem's non-convexity or from a learning-rate/conditioning issue.
Code & Formula
# Day 29 — Gradient Descent / Convex 직관
# 볼록함수 f(x) = (x - 3)^2 위에서 경사하강법을 돌려 최소점 x=3 으로 수렴하는 과정을 본다.
def f(x):
return (x - 3) ** 2
def f_prime(x):
return 2 * (x - 3)
x = 10.0 # 시작점
learning_rate = 0.1
history = [x]
for step in range(30):
grad = f_prime(x)
x = x - learning_rate * grad
history.append(x)
print("경사하강 진행 (x, f(x)):")
for i in [0, 1, 2, 5, 10, 20, len(history) - 1]:
xi = history[i]
print(f" step {i:2d}: x = {xi:.6f}, f(x) = {f(xi):.8f}")
print(f"\n최종 x = {history[-1]:.6f} (참값 x* = 3)")
print(f"최종 f(x) = {f(history[-1]):.10f} (참값 f(x*) = 0)")
Exercise
Construct a 2D quadratic function with a deliberately poor condition number, run both plain and momentum-based gradient descent on it across several learning rates, and compare the iteration counts and trajectories visually.
Practical Connection
The LMSR cost function is convex, so properties like well-defined prices and a bounded loss follow directly from convexity, and this same optimization toolkit is exactly what you use when calibrating parameters against real data.
Where it lands in Jayverse
- Verex: calibrate the market maker's liquidity parameter with a learning rate bounded by 1/L, not by trial and error. Confirm the chosen cost function (LMSR or otherwise) is actually convex before blaming the optimizer for a failure to converge.
- DeFi: when liquid-staking or rebalancing parameter calibration zig-zags, diagnose conditioning before redesigning. Add momentum or preconditioning first, and only conclude the objective is non-convex after checking that.
- Auditor: log which failure mode a calibration hit — divergence, slow convergence, or zig-zag. Each points to a different fix (learning rate too high, too low, or poor conditioning), and the write-up should name which one, not just "it didn't converge."
Key expressions
| Expression | 뜻 · 쓰이는 자리 |
|---|---|
| exploit the fact that | ~라는 사실을 이용하다 · 경사가 가장 가파른 방향을 가리킨다는 성질을 활용함 · "exploits the fact that the gradient... points in the direction" |
| steepest ascent | 가장 가파르게 오르는 방향(최대 상승 방향) · 경사의 방향을 설명하는 표현 · "the direction of steepest ascent" |
| guarantee convergence | 수렴을 보장하다 · 학습률을 특정 값 이하로 두면 수렴이 보장됨 · "choosing a learning rate at or below 1/L guarantees convergence" |
| zig-zagging | 지그재그로 왔다갔다 하는 것 · 좁고 긴 골짜기에서 최적화가 갈지자로 움직이는 현상 · "poor conditioning causes zig-zagging in narrow, elongated valleys" |
| poor conditioning | 조건수가 나쁜 상태 · 골짜기가 좁고 길어 최적화가 어려운 상황을 가리킴 · "poor conditioning causes zig-zagging" |
| mitigate | 완화하다, 줄이다 · 모멘텀·적응적 학습률 등으로 지그재그를 완화함 · "techniques like momentum... are used to mitigate it" |
| follow directly from | ~로부터 바로(당연히) 도출되다 · 볼록성만으로 여러 좋은 성질이 자동으로 성립함 · "properties like well-defined prices... follow directly from convexity" |
| L-Lipschitz | 립시츠 연속(L-Lipschitz continuous) · 기울기 변화 속도가 상수 L로 제한됨을 뜻하는 수학적 성질, 수렴 보장의 전제조건. "whose gradient is L-Lipschitz continuous" |
| LMSR | 로그 마켓 스코어링 룰(Logarithmic Market Scoring Rule, LMSR) · 예측시장에서 쓰이는 볼록 비용함수 기반 가격 결정 메커니즘. "The LMSR cost function is convex" |
| condition number | 조건수(condition number) · 골짜기가 얼마나 좁고 길쭉한지를 나타내는, 최적화 난이도의 척도. "a deliberately poor condition number, run both plain and momentum-based gradient descent" |
If you study this on a given day, add a note link and a ✅ to this line in the source curriculum (docs/knowledge/math-50-curriculum.md) and this spot will lead straight to the note body. You can also write directly on this page — but regenerating overwrites it, so it's safer to keep anything you want to save as markdown under docs/algorithms/.