从零打造一个没有数据库的金融级账务系统 eBay 支付账务系统架构演进之路-贾奇会员免费

Building a Financial Accounting System without a DatabaseThe Evolution of eBay Payment FASJacky Jia (贾奇)

Who Am I• Morgan Stanley - distributed computation in Fixed Income - low-latency trading system in Electronic Trading• eBay - financial accounting system

AgendaBackgroundSystem EvolutionSummary• FAS Introduction• Tech Challenges• Achievements1. Make Business Right2. No Data Loss3. Always Available4. Increase Throughput5. Speed Up Data Recovery6. Avoid Out-of-Memory7. Avoid Out-of-Disk• Potential Applications• Open Sourcing

What is FAS

What is FAS

Challenges to Build FAS • Data must be reliable • 7x24 service available• High throughput• 100% audit-able

Traditional Solution Uses DB

Why We did NOT Adopt the DB Solution• Data recovery challenge in Oracle• No other approved DB alternatives• Actually…it is not necessary to use DB

What We’ve Achieved So Far50ms p99100%3K/s in prod99.999%100%ReliabilityAuditabilityAvailabilityThroughputLatency

Our Solution Costs Way Less Money1. http://tpc.org/tpcc/results/tpcc_perf_results5.asp?resulttype=ALL&version=5%¤cyID=03 devs2M USD4.5M USD40M USD1<<<<

Iteration #1: Make Business Right

State Recovery

System Evolution1. Make Business Right – DDD and Event Sourcing2. No Data Loss3. Always Available4. Increase Throughput5. Speed Up Data Recovery6. Avoid OOM7. Avoid Running Out of Disk

Data May Get Loss when Machine Down

Iteration #2: Make Data Tolerate DC Down

System Evolution1. Make Business Right – DDD and Event Sourcing2. No Data Loss – Raft Cluster3. Always Available4. Increase Throughput5. Speed Up Data Recovery6. Avoid OOM7. Avoid Running Out of Disk

Recovery Takes Longer

Availability is the New Challenge

FAS Service is Stateful

Outsourcing Leader Election

Split Brain Problem

Iteration #3: Make Service Correctly Available

“Two-Leaders-In-One” Approach

System Evolution1. Make Business Right – DDD and Event Sourcing2. No Data Loss – Raft Cluster3. Always Available – Two-Leaders-In-One4. Increase Throughput5. Speed Up Data Recovery6. Avoid OOM7. Avoid Running Out of Disk

Throughput is Not Acceptable

Iteration #4: Use Batchig to Increase Throughput

System Evolution1. Make Business Right – DDD and Event Sourcing2. No Data Loss – Raft Cluster3. Always Available – Two-Leaders-In-One4. Increase Throughput – Batching5. Speed Up Data Recovery6. Avoid OOM7. Avoid Running Out of Disk

Data Recovery Can Take Minutes

Iteration #5: Make Data Recovery Fast

System Evolution1. Make Business Right – DDD and Event Sourcing2. No Data Loss – Raft Cluster3. Always Available – Two-Leaders-In-One4. Increase Throughput – Batching5. Speed Up Data Recovery – EAL to Catch Up6. Avoid OOM7. Avoid Running Out of Disk

Running Out of Memory

Iteration #6: Two-Tier State to Avoid OOM and Accelarate State Recovery

Not Worried About OOM Any More

Chaos Engineering to Ensure Correctness

Chaos Engineering to Ensure Correctness

System Evolution1. Make Business Right – DDD and Event Sourcing2. No Data Loss – Raft Cluster3. Always Available – Two-Leaders-In-One4. Increase Throughput – Batching5. Speed Up Data Recovery – EAL to Catch Up6. Avoid OOM – Two-Tier State7. Avoid Running Out of Disk

Running Out of Disk

Iteration #7: A Complete Lifecycle Management

Not Worried About OOD Any More

System Evolution1. Make Business Right – DDD and Event Sourcing2. No Data Loss – Raft Cluster3. Always Available – Two-Leaders-In-One4. Increase Throughput – Batching5. Speed Up Data Recovery – EAL to Catch Up6. Avoid OOM – Two-Tier State7. Avoid Running Out of Disk – Auto Archiving

Evolution Goes On1. Make Business Right – DDD and Event Sourcing2. No Data Loss – Raft Cluster3. Always Available – Two-Leaders-In-One4. Increase Throughput – Batching5. Speed Up Data Recovery – EAL to Catch Up6. Avoid OOM – Two-Tier State7. Avoid Running Out of Disk – Auto Archiving8. Scale Up9. Scale Out10.Read Module

Use Scenarios that can Benefit from this DesignData QualityAvailabilityLowHighLowHighFAS

FAS Infra has been Open Sourcedhttps://github.com/eBay/Gringofts

运营动脉关于我们ABOUT US WWW.YYDM.CN运营动脉是一个精心管理了4年的运营资料库，目前拥有超过40000+份运营资料，1000+运营渠道合作资源。当你在日常工作中，可以用它快速搜方案、看案例、读报告、做课件、找模板、下素材、用工具、学课程等。90%常见的运营需求都可找到模板和案例，工作中即可拿来直接套用，提高我们的工作效率，资料库持续更新ing…运营动脉官方服务号运营资料铺官方公众号运营动脉官方网站运营动脉APP下载业务联系站长微信