I conducted an a/b test but I was confused about how to interpret the result. I got 0.5% difference between control and treatment groups and statistical significance for the difference (p <0.05). In my opinion, I can only say that the two groups are different (the change effects treatment group significantly) but how can I interpret the 0.5% difference in observation? Can I say that there is significantly 0.5% difference? When it comes to null hypothesis is "A and B has no difference and I rejected it with p value, I cannot assume how different those two groups are? Then, how should I confirm "~% lift" through A/B test?
You are correct that a significant null hypothesis result only tells you that A and B are different, but it doesn't quite tell you how different they could reasonably be, given the sample size and variability in addition to the observed (sample) difference of 0.5%.
To determine a range on values the population difference could reasonably be, calculate the confidence interval. This will yield a result like, "With 95% certainty, the actual difference between A and B is between 0.28% and 0.79%." The math for confidence intervals is closely related to null hypothesis testing, so if results are statistically significant at the 0.05 level, then the lower bound of the 95% confidence interval should be greater than zero.
Here're a couple ways of calculating confidence intervals with R for the kind of data in A/B testing.
Obviously, of course, you already know the actual difference is most likely about 0.5%. Confidence intervals let you rigorously quantify how big "about" is in your instance.
I am not a statistician, but if you want to more easily see the difference between two values, use the actual difference between your results and not percentage difference.
E.g. "There were 2000 more users doing X than Y" should show a greater difference than "There were 0.05% more users doing X over Y".