DescriptionThe subject is investigating the D-brane engineering of the surface defect. First, we investigate the duality between instanton counting and refined topological string in the presence of surface defect. We construct a supersymmetric quantum mechanical model with surface operator in five dimensional SU(r) gauge theory by D-brane engineering. Then we present a conjecture formula relating the K-theoretic partition function to the refined topological amplitude. Second, we can use it as a tool to study the knot invariant. Surface operator can be engineered by toric brane in A-model topological string while A-model topological string with several toric branes on a conifold can be related to refined HOMFLY polynomial. Then we can explore the refined HOMFLY polynomial in knot theory with the help of surface operator. The formula of refined HOMFLY polynomial from physics argument is presented and it agrees with Oblomkov-Shende conjecture.